The many dimensions of truthfulness: Crowdsourcing misinformation assessments on a multidimensional scale

Recent work has demonstrated the viability of using crowdsourcing as a tool for evaluating truthfulness public statements. Under certain conditions such as: (1) having balanced set workers with different backgrounds and cognitive abilities; (2) an adequate mechanisms to control quality collected data; (3) coarse grained assessment scale, crowd can provide reliable identification fake news. However, news are subtle matter: statements be just biased (“cherrypicked”), imprecise, wrong, etc. unidimensional truth scale used in existing cannot account differences. In this paper we propose multidimensional notion ask assess seven dimensions selected based on literature: Correctness, Neutrality, Comprehensibility, Precision, Completeness, Speaker’s Trustworthiness, Informativeness. We deploy ensure that thousands assessments 180 publicly available fact-checked distributed over two datasets quality, including custom search engine by find web pages supporting their assessments. A comprehensive analysis crowdsourced judgments shows that: when compared expert-provided gold standard; proposed capture independent pieces information; task easily learned workers; (4) resulting useful basis more complete estimation statement truthfulness.